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PREFACE 


In  the  fall  of  1996,  RAND  researchers  conducted  a  successful  proof-of- 
concept  demonstration  for  the  Defense  Advanced  Research  Projects 
Agency  (DARPA)  of  new  methods  for  science  and  technology  (S&T) 
planning.  These  methods  exploit  new  information  technologies  in  order  to 

•  Improve  the  quality  of  the  S&T  planning  process  by  using  online 
computer-assisted  decision-support  tools  drat 

—  make  relevant  information  more  easily  available  to  expert 
panelists  meeting  over  the  World  Wide  Web 

—  help  panelists  include  and  compensate  for  the  effects  of 
imcertainty  by  treating  an  S&T  plan  as  a  portfolio  of 
technology  investments  hedged  against  a  wide  range  of 
plausible  futures. 

•  Provide  a  lasting  electronic  archive  of  the  expert  discussions  and 
other  information  that  go  into  building  an  S&T  plan. 

•  Lower  the  travel-related  costs  and  scheduling  constraints  of  the 
current  planning  process  by  supplementing  face-to-face  expert 
panel  meetings  with  asynchronous  meetings  over  the  World  Wide 
Web. 

The  demonstration  made  use  of  two  new  RAND  methodologies: 
HyperForum,  a  facilitated  Web-based  collaborative  exercise,  conducted  in  a 
carefully  crafted,  information-rich,  online  environment,  and  Exploratory 
Modeling,  a  new  approach  to  generating  systematic,  quantitative 
comparisons  among  alternative  policy  decisions  without  rel5dng  on 
imperfect  predictions  of  the  future. 

This  documented  briefing  describes  that  demonstration.  It  should  be  of 
interest  to  decisionmakers  responsible  for  choosing,  managing,  and 
justifying  portfolios  of  science  and  technology  research  projects  in  the 
Department  of  Defense  and  other  federal  agencies,  and  in  the  private 
sector.  In  addition,  it  should  be  informative  to  those  readers  interested  in 
how  new  information  technologies  can  provide  new  methods  for  planning 
and  decisionmaking  under  rmcertainty.  Readers  interested  in  viewing  the 
Web  site  described  in  this  documented  briefing  are  invited  to  e-mail  one  of 
the  authors:  Robert_Lempert@rand.org  or  James_Bonomo@rand.org. 

This  research  was  conducted  for  DARPA  within  the  Acquisition  and 
Technology  Policy  Center  of  RAND's  National  Defense  Research  Institute, 
a  federally  funded  research  and  development  center  sponsored  by  the 
Office  of  the  Secretary  of  Defense,  the  Joint  Staff,  the  imified  commands, 
and  the  defense  agencies. 
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1.  INTRODUCTION 


This  briefing  describes  a  proof-of-concept  demonstration  of  some  new 
methods^  for  science  and  technology  (S&T)  planning.  RAND  researchers 
conducted  this  demonstration  for  the  Defense  Advanced  Research  Projects 
Agency  (DARPA)  in  fall  1996  to  show  that  exploiting  new  information 
technologies  can  supplement  the  current  S&T  planning  process  in  useful 
ways.  In  particular,  it  can 

•  make  relevant  information  more  easily  available  for  participants  m 
the  planning  process 

•  address  imcertainty  explicitly 

•  lower  some  of  the  costs  associated  with  the  process,  especially 
travel-related  costs. 

Additionally,  the  inherent  characteristic  of  the  approach — ^using  Web- 
based  technology — ^provides  a  lasting  record  of  the  discussion  and  allows 
later  users  of  the  S&T  plan  a  much  fuller  imderstanding  of  the  basis  for 
that  plan  than  is  possible  for  paper-based  methods. 


lA  method  is  an  orderly  procedure  for  doing  something.  This  report  describes  two 
methods,  H5^erForum  and  Exploratory  Modeling.  They  can  be  used  individually  or 
together.  In  the  work  described  in  this  documented  briefing,  we  used  them  togedier. 
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This  document  presents  the  results  of  our  experiment  and  argues  that  the 
two  methods  used  in  our  approach — ^HyperForum  and  Exploratory 
Modeling — ^would  be  useful  for  the  Department  of  Defense's  S&T 
planning — in  particular,  for  future  roimds  of  the  Joint  Warfighting  Science 
and  Technology  (JWS&T)  planning  process.  That  process  considers 
primarily  the  large-scale  demonstrations  of  new  technology  within  the 
Department  of  Defense  and,  as  such,  has  both  a  continuing  need  for 
planning  and  a  need  to  convey  those  plans  to  many  audiences. 


NORIgg 


Robust  Planning  for  Defense  S&T 
Is  Difficult 


•  We  cannot  make  accurate  predictions  of 

-  Future  military  missions 

-  Future  technological  capabilities 

•  Yet  the  Director  of  Defense  Research  and 
Engineering  (DDR&E)  must 

-  Ensure  that  his  or  her  S&T  plans  are  robust 

-  Demonstrate  that  robustness  to  various 
stakeholders 

•  Current  planning  tools  do  not  support 
building  robust  plans 


randmoi 


Uncertainty  complicates  S&T  planning.  We  all  know  that  the  new  national 
security  environment  is  highly  uncertain.  We  do  not  know  the  future  uses 
of  U.S.  military  forces  or  the  capabilities  that  new  technology  may  provide. 

In  principle,  we  know  how  to  handle  such  uncertainty  in  the  S&T  planning 
activities  of  the  Department  of  Defense:  Since  S&T  plans  stretch  far  into 
the  future  and  face  great  mtemal  imcertainty,  an  S&T  plan  should  be 
considered  a  portfolio  of  technology  investments  that  is  robusf^ — ^hedging 
against  a  wide  range  of  possible  futures  and  well  positioned  to  take 
advantage  of  unexpected  opportunities. 

But  traditional  methods  of  S&T  planning  do  not  help  in  realizing  this 
vision.  Despite  best  intentions,  S&T  plans  usually  emphasize  some  best- 
guess  estimate  of  the  future.  Most  of  the  attention  and  effort  are  focused 
there,  because  that  is  seen  as  the  most  likely,  and  thus  most  important, 
case.  It  is  thus  difficult  to  assess  whether  these  plans  are  robust  against 
other  plausible  futures. 

In  addition,  current  S&T  plans  are  very  long  and  complex  documents.  The 
JWS&T  Plan^  is  two  long  volumes  with  numerous  charts  and  graphs;  the 


2  In  relation  to  a  DoD  science  and  technology  plan,  a  robust  plan  is  one  that  will  provide 
the  U.S.  military  with  aU  the  most  important  capabilities  it  will  need,  no  matter  what  type 
of  operations  it  finds  itself  conducting  in  the  future. 

^  Under  Secretary  of  Defense  (Acquisition  and  Technology)  and  the  Joint  Staff,  Joint 
Warfighting  Science  and  Technology  Plans,  Washington,  D.C.:  Department  of  Defense,  1996. 
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Defense  Technology  Area  Plan^  (DTAP)  is  longer  still  (6  volumes).  Even  if 
the  S&T  plans  are  robust,  it  is  difficult  to  use  such  documents  to 
demonstrate  that  robustness  to  distant,  but  still  interested,  stakeholders 
such  as  Congress. 

The  current  process  for  creating  the  defense  S&T  plans  and  the  related 
documents  is  also  labor-intensive.  Like  many  S&T  planning  processes,  it 
fundamentally  relies  upon  expert  opinion.  Through  a  series  of  meetings,  a 
group  knowledgeable  about  &e  technical  area  reaches  some  consensus  on 
the  future  prospects  and  the  appropriate  future  plans. 

The  experiment  described  in  this  document  involved  only  a  part  of  the 
overall  defense  S&T  planning  process,  that  involving  tihe  Information 
Superiority  segment  of  the  Joint  Warfighting  Science  and  Technology  Plan 
(JWS&TP).  For  imderstanding  the  implications  of  our  experiment,  it  is 
important  to  tmderstand  the  basics  of  the  existing  process  for  generating 
that  plan. 

The  process  begins  with  a  set  of  Joint  Warfighting  Capability  Objectives 
QWCOs)  developed  by  the  Joint  Staff  and  coordinated  with  the  Office  of 
the  Secretary  of  Defense  (OSD)  and  with  the  Service  science  and 
technology  executives.  The  JWCOs  are  not  intended  to  be  a 
comprehensive  set  of  objectives  that  cover  all  the  demands  of  the 
Department  of  Defense  for  advanced  technologies.  Rather,  they 
"represent  some  of  the  most  critical  capabilities  for  maintaining  the 
warfighting  advantage  of  U.S.  forces"  (JWS&TP,  Chapter  II,  "Vision  and 
Strategy,"  at  http://www.dtic.mil/dstp/DSTP/97_jwstp/jw2.htm  on 
October  31, 1997).  The  JWCOs  change  in  number  and  content  over  time. 
Information  Superiority  has  consistently  been  an  important  JWCO, 
probably  because  of  its  obvious  importance  for  future  military  dominance 
in  most,  if  not  all,  types  of  future  operations. 

For  each  JWCO,  a  panel  is  created  rmder  the  oversight  of  the  Defense  S&T 
Reliance  Executive  Committee  (EXCOM).  The  EXCOM  is  chaired  by  the 
Deputy,  DDR&E,  and  otherwise  includes  the  Deputy  Assistant  Secretary 
of  the  Army  (Research  and  Technology);  the  Chief  of  Naval  Research;  the 
Deputy  Assistant  Secretary  of  the  Air  Force  (Science,  Technology,  and 
Engineering);  the  Deputy  Director,  Defense  Advanced  Research  Projects 
Agency;  the  Assistant  Deputy  Director  for  Technology,  Ballistic  Missile 
Defense  Organization;  and  the  Deputy  Director,  Defense  Special  Weapons 
Agency. 

The  panel  for  each  JWCO  is  made  up  primarily  of  technical  experts  in  the 
subject  matter  of  the  JWCO.  It  is  chaired  by  an  individual  selected  by  the 


*  Director,  Defense  Research  and  Engineering,  Defense  Technology  Area  Plan,  Washington, 
D.C.:  Department  of  Defense,  1996. 


Office  of  the  Director  of  Defense  Research  and  Engineering  (ODDR&E), 
and  includes  the  membership  of  all  the  above  organizations  that  have 
relevant  expertise,  plus  representatives  from  the  Joint  Staff.  These  panels 
hold  face-to-face  meetings  to  discuss  and,  eventually,  to  fashion  a  list  of 
future  demonstrations — the  Advanced  Technology  Demonstrations 
(ATDs)  and  the  Advanced  Concept  Technology  Demonstrations  (ACTDs). 
These  demonstrations  are  then  used  to  create  Defense  Technology 
Objectives  (DTOs)  for  each  demonstration. 

It  is  important  to  realize  that  these  panels  rely  almost  entirely  upon  the 
expertise  and  knowledge  the  panelists  bring  to  the  deliberations.  While 
these  panels  can,  and  do,  draw  upon  relevant  analytic  work,  they  have  too 
little  time  and  support  to  embark  on  analysis  themselves.  This  leaves  their 
expert  opinions  as  the  key  input  to  their  decisions. 

After  these  plans  are  created,  a  group  outside  the  Department  of  Defense 
reviews  each  technical  area  to  ensure  a  wide  perspective  on  technical 
developments.  This  is  the  Technology  Area  Review  and  Assessment 
(TARA)  process.  The  TARA  teams  must  have  at  least  two-thirds  of  their 
membership  from  outside  DoD.  "Most  TARA  team  members  are 
recognized  experts  from  the  National  Academy  of  Sciences,  the  National 
Academy  of  Engmeering,  the  Institute  of  Medicine,  the  Defense  Science 
Board,  the  scientific  advisory  boards  of  the  military  departments,  industry, 
and  academia.  The  TARA  team  is  chaired  by  a  senior  executive  appointed 
by  the  DDR&E.  The  appropriate  representatives  from  the  Defense  S&T 
Reliance  Technical  Panel  brief  the  DoD  program  as  compared  to  the 
planning  guidance"  QWS&TP,  Chapter  II,  "Vision  and  Strategy,"  at  http:/ 
/www.dtic.mil/dstp/DSTP/97_jwstp/jw2.htm  on  October  31, 1997).  The 
results  of  the  TARA  process  are  filtered  through  the  DDR&E-chaired 
Defense  Science  and  Technology  Advisory  Group  (DSTAG),  and  the 
DDR&E  ultimately  submits  suggested  changes  to  the  Department  of 
Defense  budget  decisionmaking  process. 

Our  experiment  was  designed  within  this  overall  framework.  That  is,  we 
too  used  expert  panels  and  relied  upon  the  panels'  understanding  of 
operational  needs  and  of  technical  possibilities.  WhUe  it  would  be 
possible  to  use  the  S&T  planning  methods  described  in  this  document  to 
link  such  panels  to  numerous  existing  studies  or  even  to  tools  allowing  the 
panels  to  do  technical  analysis,  that  was  not  our  aim  here.  Instead,  we 
focused  on  expanding  and  enhancing  the  existing  methodology  of  these 
panels,  not  on  changing  their  fundamental  reliance  on  expert  judgment. 
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An  Experiment  Indicated  a  Solution 


Thesis: 

•  New  information  technologies  can  provide  DoD  with  analytic  S&T 
planning  methods  having  previously  unavailable  capabilities 

•  Methods  can  ease  many  constraints  of  traditional  planning 
Experiment: 

•  Use  Web-based  HyperForum  to  gather  expert  opinions 

•  Use  Exploratory  Modeling  to  determine  whether  S&T  strategies  are 
robust  across  many  futures 

Result: 

•  Experiment  demonstrated  feasibility  of  method 

•  Methods  affected  planning  process  itself 

mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm  randmmi 


Our  h5rpothesis  was  that  new  information  technology  can  improve  the 
current  S&T  planning  process  by  providing  DoD  with  analytic  planning 
methods  that  have  previously  imavailable  capabilities  (James  Bonomo, 
"Tools  for  Strategic  Planning  at  DARPA/'  tmpublished  RAND  research). 

We  employed  two  main  techniques,  or  methods.  First,  we  used  the  Web  to 
let  expert  panels  work  online,  reducing  the  need  for  face-to-face  meetings 
and  increasing  the  information  readily  available  to  them.  Second,  we  used 
Exploratory  Modeling  to  enable  tiie  panelists  to  consider  a  large  number  of 
future  scenarios. 

The  experiment  worked.  We  showed  that  these  methods  can  make 
valuable  contributions  to  S&T  planning.  We  also  helped  our  panelists 
come  to  a  deeper  understanding  of  the  plans  they  were  producing. 

Most  important,  the  experiment  showed  that  a  robust  plan  could  be 
constructed,  and  that  the  robustness  of  that  plan  could  be  conveyed  to  at 
least  some  people  who  were  outside  the  planning  process. 
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A  HyperForum  Helps  Groups  Work 
Together  on  Complex  Policy  Problems 

What  is  a  HyperForum? 

•  a  facilitated,  Web-based  collaborative  exercise,  conducted  in  a 
carefully  crafted,  information-rich,  online  environment 

•  a  set  of  techniques  and  technology  being  developed  by  RAND, 
Caltech,  and  the  Markle  Foundation. 

Using  the  Web,  groups  developing  technology  plans  can 
draw  on  and  convey  more  information: 

•  Databases  can  be  searched  and  manipulated 

•  Nodes  can  be  linked  to  other  relevant  sources  of  information 

•  Forums  are  available  to  facilitate  inputs  and  buy-in  from  many 
stakeholders 


RAND 


Before  we  describe  our  experiment,  we  need  to  describe  the  two  methods — 
H^yperForum  and  Exploratory  Modeling. 

In  addition  to  the  S&T  planning  HyperForum  conducted  for  DARPA,  we  also 
conducted  a  HyperForum  on  Sustainability  in  collaboration  with  Caltech,  the 
World  Resources  Institute,  and  the  Santa  Fe  Institute,  in  a  project  funded  by 
die  Markle  Foimdation  (see  "Internet  Changing  How  Research  Centers 
Work,"  New  York  Times,  Jime  9, 1997,  p.  D5,  in  the  Appendix).  The  name 
H5rperForum  is  a  registered  trademark  owned  by  the  Markle  Foundation. 

A  H5q>erForum  is  a  facilitated,  Web-based  collaborative  exercise,  conducted 
in  a  carefully  crafted,  information-rich,  online  environment.^  It  is  helpful  to 
differentiate  the  HyperForum  idea  from  the  many  other  online  conferencing 
efforts  that  are  imder  way.  In  contrast  to  newsgroups,  chat  rooms,  and  group 
work  over  intranets,  a  H5^erForum  is  heavily  facilitated,  embeds,  on  the  site. 


®  Most  current  Web  site  design  seeks  to  make  as  much  imfiltered  information  available  to 
users  as  possible,  and  the  amount  of  information  can  rapidly  overwhelm  those  users.  Rather 
than  using  such  automation  as  search  engines,  autonomous  agents,  or  advanced  site  maps  to 
prevent  this  problem,  we  adopted  a  more  labor-intensive  approach  that  we  believe  is  more 
appropriate  for  some  policy  applications.  We  carefully  chose  and  limited  the  information  we 
placed  on  the  site  to  that  information  the  panelists  would  find  most  useful  in  addressing  the 
tasks  we  gave  them. 

The  discussions  are  facilitated  by  a  person  in  much  the  same  way  a  face-to-face  meeting  or 
conference  is  facilitated  by  the  chairperson. 
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a  great  deal  of  customized  information  for  the  panelists,  and  organizes  the 
site  using  a  task-oriented  rather  than  a  browsing-oriented  model.^ 
HyperFora  draw  heavily  on  commercially  available  technology  (e.g., 
discussion  tools,  graphics  tools),  although  we  have  developed  some 
specialized  tools — ^in  particular,  custom  software  required  for  individual 
HyperFora. 

The  HyperForum  method  helps  the  technology-plarming  process  draw  on 
and  convey  more  information.  In  particular,  the  plans  themselves  become 
databases  that  can  be  searched,  manipulated,  and  Unked  to  other  relevant 
documents.  And  the  planning  process  can  be  opened  to  inputs  and  buy-in 
from  many  more  stakeholders  than  could  attend  face-to-face  meetings. 


^  Much  like  a  library,  most  Web  sites  are  browser-oriented  in  that  their  creators  are  neutral 
about  what  information  a  user  will  seek  and  in  what  order. 

A  HyperForum  is  like  a  face-to-face  meeting  in  that  it  is  designed  to  help  the  panelists 
achieve  a  specific  purpose.  Therefore,  the  information  made  available  when  the  panelists  are 
at  the  HyperForum  site  is  governed  by  a  temporal  agenda  and  a  specific  set  of  procedures. 
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Exploratory  Modeling  Helps  Make 
Systematic  Choices  Without  Predictions 

•  What  is  Exploratory  Modeling? 

-  A  new  approach  to  generating  systematic,  quantitative 
comparisons  among  alternative  policy  decisions  that 
combines  scenario-planning  methodologies  with  quantitative 
forecasting  techniques 

-  Enabled  by  today’s  information  technology 

-  Applied  in  part  here;  more  fuiiy  elsewhere^ 


•  With  Exploratory  Modeling,  S&T  plans  can  address  uncertainty: 

-  Find  robust  combinations  of  plans  or  identify  holes  by  looking 
at  patterns  in  the  visualizations 


-  View  technology  plans  as  portfolios  that  take  advantage  of 
fortuitous  opportunities  and  hedge  against  adverse  surprises 

-  Consider  adaptive  plans  explicitly  designed  to  respond  as  new 
information  becomes  available 

RANDMBi 

•R.  J.  Lempert,  M.  E.  Schlesinger,  and  S.  C.  Bankes,  "When  We  Don't  Know  the  Costs  or  the  Benefits: 

Adaptive  Strategies  for  Abating  Climate  Change,”  Climatic  Change  33: 235-274  (1996). 


Exploratory  Modeling,  or  xM,  is  the  second  technology  we  used  in  our 
experiment.  A  new  approach  to  generating  systematic,  quantitative 
comparisons  among  alternative  policy  decisions  without  relying  on 
imperfect  predictions  of  the  the  future,  xM  marries  today's  information 
technology — ^networked  computer  workstations  and  powerful  graphics — 
to  a  new  concept  for  supporting  decisionmaking  xmder  xmcertainty.  It 
combines  the  wide-ranging  scope  of  scenario-planning  methodologies 
with  the  strengths  of  the  best  quantitative  forecasting  techniques. 

The  basic  concept  of  xM  is  to  create  a  database  of  many  (tens  to  millions) 
scenarios  and  then  use  computer  search  and  visualization  techniques  to 
imderstand  the  implications  these  scenarios  have  for  policy  choices.  This 
briefing  presents  an  application  with  only  a  very  small  number  of 
scenarios,  but  the  methods  can  be  used  to  handle  several  orders  of 
magrutude  more  scenarios,  as  we  are  already  doing  in  other  policy  areas, 
such  as  our  work  on  climate  change  (R.  J.  Lempert,  M.  E.  Schlesinger,  and 
S.  C.  Bankes,  "When  We  Don't  Know  the  Costs  or  the  Benefits:  Adaptive 
Strategies  for  Abating  Climate  Change,"  Climatic  Change  33:  235-274, 1996) 
and  weapons  procurement  (Arthur  Brooks,  Steve  Bankes,  and  Bart 
Bennett,  Weapon  Mix  and  Exploratory  Analysis:  A  Case  Study,  Santa  Monica, 
CA:  RAND,  DB-216/2-AF,  1997). 

Exploratory  Modeling  helps  the  S&T  planning  process  address  uncertainty 
explicitly,  identif3Tng  holes  or  finding  robust  combinations  by  looking  at 
patterns  in  visualizations.  In  addition,  we  showed  how  these  methods 
help  treat  technology  plans  as  portfolios  that  take  advantage  of  fortuitous 
opporhmities  and  hedge  against  adverse  surprises. 
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Comparison  with  Other  Methods 


Exploratory  Modeling  draws  upon  and  combines  the  benefits  of  many 
previous  methods  for  making  daoices  about  the  future.  We've  chosen 
to  compare  these  methods  along  three  dimensions:  the  degree  to  which 
a  method  uses  quantitative  models  to  describe  the  system,  the  degree  to 
which  it  considers  multiple  futures,  and  the  degree  to  which  it  involves 
multiple  stakeholders  or  experts. 

Let's  start  along  the  quantitative-model  axis.  Quantitative  tools  are 
important  because  they  address  "how  much"  questions  (How  much 
should  a  firm  spend  on  a  new  technology?  What  growth  rate  indicates  a 
new  product  is  a  success?)  and  can  yield  imexpected  insights  about  the 
strengths  and  weaknesses  of  different  decisions. 

Delphi  is  the  first  method  along  this  axis.  It  uses  iterated,  anonymous 
polling  of  experts,  but  uses  no  quantitative  model  and  yields  only  one 
consensual  view  of  the  future.  Forecasting  techniques,  such  as 
technology  forecasts  or  market  forecasts,  employ  some  variation  of 
trend  extrapolation.  Forecasting  can  involve  multiple  stakeholders  but 
employs  quantitative  models  designed  to  yield  a  single  "best  estimate" 
about  the  future.  "Traditional  tools"  is  our  label  for  the  elegant 
mathematical  machinery  built  aroimd  optimization.  These  tools 
assume  static  outcomes,  even  for  inherently  dynamic  processes,  and 
then  maximize  those  outcomes.  This  class  includes  much  of  operations 
research,  decision  theory,  and  the  cost-benefit  tools  of  neoclassical 
economics.  These  techniques  generally  involve  one  research  team  and 
consider  one  future. 
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Some  variant  of  these  methods — ^Delphi,  forecasting,  and  optimization — 
underlies  most  prediction-based  policy  analysis. 

There  are  also  several  methods  for  thinking  about  multiple  futures. 
Assumption-Based  Planning  is  a  RAND-developed  scenario-planning 
method  (James  A.  Dewar,  Carl  H.  Builder,  William  M.  Hix,  and  Morlie  H. 
Levin,  Assumption-Based  Planning:  A  Planning  Tool  for  Very  Uncertain  Times, 
Santa  Monica,  CA:  RAND,  MR-114-A,  1993)  that  is  somewhat  similar  to 
the  scenario-planning  techniques  most  famously  employed  by  Royal 
Dutch  Shell  and  by  Peter  Schwartz  and  his  Global  Business  Network  (Peter 
Schwartz,  The  Art  of  the  Long  View,  New  York:  Doubleday/Cxurency, 

1991).  Scenario  planning  usually  involves  the  members  of  a  commimity  or 
organization  in  developing  a  small  number  of  compelling  scenarios  that 
reveal  the  key  assumptions  underlying  the  group's  plans.  Such  methods 
focus  on  multiple  stakeholders  and  multiple  futures  but  generally  have  no 
quantitative  model. 

CAS,  or  complex  adaptive  systems,  is  often  associated  with  the  Santa  Fe 
Institute.  CAS  uses  highly  nonlinear  mathematics  to  capttue  the 
xmpredictability,  evolution,  and  adaptability  of  systems  made  of  many 
interacting  agents.  We  show  CAS  models  as  being  more  quantitative  than 
optimization,  because  ttie  latter  makes  many  assumptions  about  linearity 
that  are  often  not  true.  CAS  models  will  usually  generate  many  futvures. 
The  main  problem  with  CAS  is  that  no  one  has  yet  figured  out  how  to 
make  usefol  comparisons  among  potential  decisions  using  this  fascinating 
mathematics. 

Finally,  we  show  Exploratory  Modeling  as  building  on  the  best  ideas  in  the 
other  methods.  Exploratory  Modeling  can  employ  the  quantitative  models 
from  the  forecasting,  optimization,  and  CAS  methods.  It  can  use  the 
models  within  the  multi-scenario,  organizational  planning  process  used  in 
the  scenario-based  planning  methods. 

The  entire  region  labeled  Exploratory  Modeling  on  this  chart  has  not  yet 
been  filled  in  with  examples  of  successful  studies.  What  we  have  done  so 
far  is  contained  in  the  back  reaches  of  this  space. 
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Flow  of  Experimental  Planning  Process 


MBINDRI 


Task  1: 

Task  2: 

Task  3: 

Task  4: 

Task  5: 


Create  Web  site  with 
specialized  tools  and 
content 

Convene  expert  panels 


Conduct  Exploratory 
Modeling  exercise 

Iterate  with  expert 
panels 

Provide  decisionmakers  with 
plan  and  history 

wmmmmmmmm  randmi 


This  chart  shows  how  we  orgaitized  our  experiment  for  DARPA, 
using  HyperForum  and  Exploratory  Modeling.  The  experiment 
follows  the  flow  of  an  actual  planning  process. 

In  this  demonstration  project,  we  worked  through  only  the  first  three 
steps  because  of  funding  limitations.  Tasks  1  tiirough  3  are  described 
in  Sections  2  through  4.  Section  5  discusses  the  results  of  the 
experiment  and  offers  conclusions. 
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Flow  of  Experimental  Planning  Process 


NDRI 


Task  1: 

Task  2: 

Task  3: 

Task  4: 

Task  5: 


Create  Web  site  with 
specialized  tools  and 
content 

Convene  expert  panels 


Conduct  Exploratory 
Modeling  exercise 

Iterate  with  expert 
panels 

Provide  decisionmakers  with 
plan  and  history 


2.  TASK  1:  CREATE  WEB  SITE  WITH 
SPECIALIZED  TOOLS  AND  CONTENT 
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NDRI 


Web  Site  Built  from  the 
JWS&T  Plan  and  ABIS 


•  Both  efforts  link  from  JWCO  to 
technology  demonstrations 

•  Plans  already  supply  significant 
programmatic  information 

•  Efforts  have  significant  acceptance  in 
DoD 


randhm 


In  the  first  task,  creating  our  Web  site,  we  put  online  the  Information 
Superiority  section  of  the  then-current  JWS&T  Plan  (Joint  Warfighting 
Science  and  Technology  Plan,  1996;  Defense  Technology  Area  Plan,  1996).  We 
focused  on  the  Information  Superiority  section,  because  we  had  become 
familiar  with  that  part  of  the  plan  when  RAND  helped  DARPA  with  the 
Advanced  Battlespace  Information  System  (ABIS)  study,  which  provided 
much  of  both  the  structure  and  the  substance  for  the  Information 
Superiority  section  of  the  JWS&T  Plan.^ 

In  particular,  botih  the  ABIS  study  and  the  JWS&T  Plan  rely  upon  an 
explicit  connection  between  operational  effectiveness  and  technology.  The 
connection  uses  ffie  expert  but  subjective  opinion  of  the  panel  building  the 
plan.  The  panel  focused  on  the  comparatively  well-defined  Joint 
Warfighting  Capability  Objective  for  Information  Superiority  and 
estimated  how  particular  demonstrations  or  technology  developments 
supported  part  of  that  JWCO.  That  connection  provided  us  witih  the 
ready-made  and  accepted  method  that  we  needed  for  our  experiment. 

Additionally,  the  JWS&T  Plan  contains  readily  available,  extensive 
information  on  DoD  S&T  programs,  making  it  a  natural  source  for 
planning  activities.  It  also  has  rapidly  become  a  widely  accepted 
document  among  participants  in,  and  the  audience  for,  our  experiment. 


7  Joint  Staff,  Director  of  Command,  Control,  Communications,  and  Computers,  and  Office 
of  the  Secretary  of  Defense,  Director,  Defense  Research  and  Engineering,  Advanced 
Battlespace  Information  System  (ABIS)  Task  Force  Report,  Volumes  I- VI,  Washington,  D.C., 
May  1996. 
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Methods  Help  improve  on 

JWS&T  Plan  and  ABIS 

. . 


•  Allocate  S&T  investments  as  hedges  against 
different  possible  futures 

•  Consider  budgetary  trade>offs  or  constraints 

•  Draw  information  from  experts  outside 
traditional  community 

•  Provide  an  audit  trail  to  trace  the  effects  of 
changes  in  assumptions 


RANDlBSSm 


We  put  a  variety  of  software  tools  on  our  Web  site  to  help  the  panelists 
examine  the  implications  of  the  JWS&T  Plan  and  explore  potential 
modifications  in  a  way  not  possible  by  working  on  paper  and  in  face-to- 
face  meetings.  We  describe  these  tools  in  more  detail  in  the  next  section. 
We  then  used  our  HyperForum  and  Exploratory  Modeling  methods  to 
demonstrate  how  the  process  of  creating  the  JWS&T  Plan  and  ABIS  could 
be  improved. 
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Flow  of  Experimental  Planning  Process 


INDRII 


Task  1: 

Task  2: 

Task  3: 

Task  4: 

Tasks: 


Create  Web  site  with 
specialized  tools  and 
content 

Convene  expert  panels 


Conduct  Exploratory 
Modeling  exercise 

Iterate  with  expert 
panels 

Provide  decisionmakers  with 
plan  and  history 

RANDam 


3.  TASK  2:  CONVENE  EXPERT  PANELS 


After  creating  our  Web  site  with  its  specialized  tools  and  content,  we  next 
convened  our  expert  panels.  Each  panelist  needed  only  a  standard  Web 
browser  (such  as  a  recent  version  of  Netscape  Navigator),  the  URL  for  our 
Web  site,  and  a  password.  AH  the  tools  for  manipulating  plans  and 
communicating  to  the  other  panelists  were  accessible  through  the  browser; 
the  (small)  computational  load  was  handled  on  our  UNIX  server.  This 
protocol  enabled  several  of  our  panelists  to  access  the  site  from  simple 
laptops  while  traveling. 
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NDRI 


Convene  Expert  Panels  Online 


•  Panels  charged  with  modifying  the  Information  Superiority 
portion  of  JWS&T  Plan  to  address  two  new  military  scenarios, 
a  “No-Warning”  MRC  and  Urban  Combat 

•  Panels  met  online  from  November  11  to  December  9, 1996 

-  Heavy  facilitation 

-  Kick-off  meeting  was  face-to-face 


•  Panel  divided  into  two  teams: 

Team  A  (No-Warning  MRC) 

Dr.  Ed  Brady 

Dr.  Paul  Davis 

Lt  Col  Tom  Fossen  (USAF) 

Lt  Col  Dan  McCusker  (USAF) 
CDR  Nicholas  Trongale  (USN) 


Team  B  (Urban  Combat) 

Dr.  Richard  Darilek 
MAJ  Steve  Galing  (USA) 

LtCol  Colin  Lampard  (USMC) 

Dr.  Dave  Ochmanek 
Dr.  John  Poindexter 
mmmmmmmmmmmmmm  randmi 


Our  expert  panel  had  ten  members:  five  military  officers,  three  members 
of  the  ]^ND  research  staff  (Davis,  Darilek,  and  Ochmanek,  who  had  just 
returned  from  the  Pentagon),  and  two  consultants  (Brady  and  Poindexter) 
nominated  by  DARPA.  We  divided  the  panel  into  two  teams  of  five  each. 
Each  team  had  a  member  of  our  project  staff  serve  as  a  facilitator. 

We  asked  each  team  to  start  with  the  Information  Superiority  portion  of 
the  JWS&T  Plan  and  modify  it  to  provide  capabilities  the  U.S.  military 
would  need  to  conduct  successful  operations  in  two  new  military  scenarios 
in  the  year  2010 — a  No-Waming  MRC  (major  regional  contingency)  and 
Urban  Combat.  Each  panel  first  met  in  an  introductory  session  at  RAND 
in  both  Santa  Monica  and  Washington,  D.C.  The  first  session  was  a  face- 
to-face  meeting.  The  participants  on  one  coast  were  connected  via 
telephone  with  the  participants  meeting  simultaneously  on  the  other  coast. 
The  rest  of  the  panel  interactions  occurred  online  over  the  course  of  one 
month. 

We  should  emphasize  that  these  panelists  were  chosen  as  part  of  a  proof- 
of-concept  exercise.  While  they  represent  a  broad  base  of  talent  and 
expertise,  they  did  not  cover  all  the  substantive  areas  we  would  have 
needed  for  an  actual  exercise  intended  to  inform  DoD  policy  choices. 
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Each  Panel  Was  Charged  with 
Modifying  the  JWS&T  Pian  in  Discrete  Steps 
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Emedl:  Te8imB@raM.Qrjg  Teclmicsd  Support:  terr^@raM.Qr^ 


We  divided  each  team's  work  into  two  phases.  This  figure  shows  the 
three-step  homepage  Team  B  saw  during  the  first  phase  of  their  work. 

Each  phase  lasted  for  a  few  days  to  a  week.  The  panelists  would  come  to 
this  page  for  instructions  and  for  an  overview  of  the  material  on  the  site. 

In  the  second  phase,  we  modified  the  homepage  as  described  below. 

We  asked  Team  B  to  modify  the  JWS&T  Plan  to  address  future  urban 
combat.  Each  of  the  three  areas  reached  by  the  links — ^labeled  Step  1,  Step 
2,  and  Step  3 — ^has  tools,  content,  and  discussion  space  for  the  task  at  hand. 
When  one  step  was  complete,  the  facilitator  would  ask  the  participant  to 
move  to  the  next  step. 

In  Step  1,  we  asked  the  team  to  develop  concepts  of  operations  for  future 
urban  combat.  The  "Military  Scenario"  link  provides  a  detailed  briefing  on 
an  urban-combat  scenario,  based  on  prior  RAND  work  0ohn  A.  Friel  and 
Bruce  W.  Don,  "Close  Support,  Scenario  and  Battlefield  Situation 
Assessment,"  impubHshed  RAND  research).  The  "Step  1"  link  provides  a 
place  for  the  panel  to  carry  on  their  discussions.  In  Step  2,  we  asked  the 
panel  to  modify  the  JWS&T  Plan  so  that  it  would  better  support  these 
concepts  of  operations.  The  "Choose  Plan"  fink  provides  a  discussion  area, 
and  the  "Team  B:  Workspace"  provides  a  variety  of  tools  for  manipulating 
various  versions  of  the  technology  plan.  In  Step  3,  we  asked  the  panel  to 
assess  how  weU  the  United  States  could  conduct  urban-combat  operations 
if  this  new  plan  were  carried  out. 
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step  1 — Operational  Concepts: 

History  of  Discussion  Is  Maintained  and  Available 
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In  Step  1,  we  asked  Team  B  to  develop  a  concept  of  operations  for  urban 
combat.  Shown  here  is  a  typical  discussion,  carried  out  using  a  share-ware 
tool  called  HyperNews.  Each  comment  has  a  title,  shows  die  author,  and 
is  positioned  in  relation  to  the  other  comments  in  the  discussion.  The 
indentations  allow  a  reader  to  follow  the  various  threads  of  the  discussion 
as  panelists  respond  to  previous  comments  or  start  a  discussion  on  a  new 
topic.  The  icons  appearing  before  each  message  title  are  a  standard  feature 
of  HyperNews.  Their  use  in  voting  is  described  on  the  chart  after  next. 

Note  that  panelists  respond  to  the  facilitator  and  to  each  other.  One 
important  role  of  the  facilitator  was  in  fact  to  keep  the  different  threads  of 
the  discussion  focused  and  relevant  to  the  panel's  overall  goals,  so  that 
when  an  important  new  idea  appeared,  the  facilitator  used  that  idea  to 
create  a  new  "thread."  Creating  new  threads  allowed  the  discussions  on 
different  issues  to  move  forward  separately,  without  becoming  confusing. 

To  ease  navigation  through  those  issues,  the  facilitator  would  post  a 
"summary"  message  from  time  to  time.  The  summary  would  be 
announced  through  a  short  e-mail  message  sent  separately  to  the 
participants.  The  posted  "summary"  message  would  draw  attention  to 
important  issues  in  other  messages,  wherever  they  had  been  posted,  and 
included  links  to  them  through  hyperlinks.  The  posted  message  also 
suggested  specific  topics  that  needed  discussion. 

The  resultant  threaded  arrangement  of  the  comments  makes  it  easy  to 
follow  the  discussion,  both  while  it  is  going  on  and  afterwards.  One 
author  (Lempert),  who  did  not  participate  in  the  panels,  worked  his  way 
through  the  discussions  after  they  were  completed  and  foimd  it 
surprisingly  easy  to  get  a  good  understanding  of  a  complicated  and 
multifaceted  debate. 
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Concepts  Are  Generated,  and 
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This  is  an  example  of  a  concept  and  the  depth  of  discussion  that  could  be 
engendered.  This  idea  was  raised  by  one  of  the  panelists  early  in  our 
discussion  of  urban  combat.  He  called  it  then  "optical  stealth"  but  quickly 
decided  "mirage"  was  closer  to  what  he  had  in  mind.  The  concept,  which 
he  developed  more  in  the  message  shown  above,  appealed  in  principle  to 
the  other  panelists.  It  seemed  a  natural  extension  of  well-established 
methods  of  urban  combat,  such  as  the  use  of  smoke. 

Naturally,  the  technical  feasibility  of  tiiis  idea  was  also  considered  later  as 
the  panel  built  an  S&T  plan.  Here,  however,  the  discussion  focused  more 
on  the  operational  implications  of  plausible  technical  capabilities.  Such 
discussions  always  mix  issues  of  feasibility  and  operational  effectiveness. 
Moreover,  in  this  H5q>erFormn,  tihe  permanence  of  the  comments  allowed 
the  facilitator  to  recall  any  comments  on  these  issues  and  bring  them, 
again,  to  the  attention  of  the  entire  group. 

Mechanically,  the  process  of  entering  a  concept,  or  responding  to  one, 
quickly  seemed  to  become  second  nature  to  the  panelists.  After  an  initial 
week  of  learning  to  use  the  HyperNews  tool,  they  responded  easily  and 
appropriately  to  the  discussion  threads. 

HyperNews  itself  facilitated  such  response  with  several  useful  featiures.  It 
provides  a  highlighted  title,  the  author's  name  (we  have  deleted  the  name 
here  because  we  promised  our  panelists  that  we  would  not  identify  diem 
publicly  with  their  comments),  and  date  of  a  message.  At  the  bottom  of 
the  message  (not  visible  in  the  image  on  the  facing  page),  buttons  that  are 
clicked  on  take  a  reader  to  a  composition  page,  where  he  or  she  can  enter 
comments.  Those  comments,  when  finished  and  submitted,  then  appear  as 
titles  below  the  message,  or  on  the  summary  of  threads,  as  in  the  preceding 
chart. 
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Example  of  Voting  on  a 
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g  linted  ^Ltdol.  OoM  Lamtiaied .  USMC  (ParrejistNo:  l4)) : 
^ Uatittedf  JoM Pokidea^tsr : ^PaMlIst Ha.  i6»  " 

I  QpMbftiMai  Giitag;  USA  ^Peaelist No.  13)) 

OVsejwr  (PsxieBst  No;  20)) 


Use  of  H5rperNews  in  this  HyperForum  evolved  during  this  short 
experiment.  One  of  the  authors,  acting  as  the  facilitator  for  the  urban- 
combat  team,  developed  a  simple  metiiod  of  voting  on  a  summary:  First, 
the  facilitator  fashioned  a  terse  statement  of  an  apparent  consensus  within 
the  group.  This  consensus  was  then  presented,  as  in  the  rectangle  on  ti\e 
facing  page,  for  the  participants  to  vote  on,  within  one  of  the  siunmary 
messages  described  earlier.  The  voting  method  simply  used  two  of  the 
already-available  icons  supplied  by  HyperNews.  A  thumbs-up  indicated 
support,  and  a  thumbs-down  indicated  disagreement.  These  icons  (which 
could  be  used  at  any  time  to  tag  any  submitted  message)  were  reserved  for 
this  purpose,  through  a  social  agreement  within  the  group  on  their  use. 

The  consequence  of  such  voting  was  that  issues  could  be  resolved  by  the 
panelists.  More  important,  each  participant  was  encouraged  to  take  a 
visible  stand  on  each  issue.  Most  attempts  at  summaries  were  in  fact 
agreed  on  by  the  panelists,  probably  because  the  facilitator  waited  until 
there  were  clear  signs  of  consensus.  At  other  times,  an  attempted 
summary  prompted  a  deeper  discussion  of  an  issue,  such  as  on  the  tension 
between  a  commander's  interest  in  "drilling  down"  for  (micromanaging) 
information  on  the  execution  of  a  mission  versus  a  subordinate's  interest  in 
avoiding  micromanagement. 

While  all  of  the  examples  have  concerned  the  first  step  of  the  planning 
process,  developing  a  concept  of  operations,  the  methods  of  facilitating  a 
discussion  described  for  this  step  were  used  in  all  later  steps  as  weU. 
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step  2 — Create  a  Technology  Plan: 

Panelists  Can  Manipulate  Technology  Plans  Online 
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Once  the  panelists  agreed  on  concepts  of  operations,  the  facilitator  asked 
them  to  move  on  to  Step  2.  Here,  we  asked  the  panelists  to  modify  the 
JWS&T  Plan  to  support  the  concepts  of  operations  developed  in  Step  1.  It 
was  important  that  this  modified  plan  fit  within  the  same  budget  as  the 
original  JWS&T  Plan.  This  step  differed  from  the  first  in  its  heavy  use  of 
online  tools. 

The  chart  on  the  facing  page  is  what  an  S&T  plan  looks  like  on  our  site. 

The  plan  itself  is  a  database  that  can  be  manipulated,  searched,  and  shared 
among  panelists.  In  the  left-most  column  is  a  Hst  of  technology  programs. 
Clicking  on  a  name  produces  a  page  with  the  programmatic  information 
provided  in  the  Appendix  of  the  JWS&T  Plan:  program  description, 
budgets,  and  a  matrix  showing  those  military  operational  capability 
elements  (OCEs)  supported  by  the  program.  The  next  columns  show 
whether  this  program  has  been  added,  deleted,  or  modified  relative  to  the 
original  JWS&T  Plan.  The  next  coliunns  show  the  budget. 

The  military-contribution  column  shows  a  rough  measure  of  the  program's 
contribution  to  military  operational  capabilities.  In  the  JWS&T  process, 
each  program  in  the  technology  plan  was  given  a  black  dot,  a  white  dot,  or 
no  dot,  representing  "strong,"  "moderate,"  or  "no  support"'  for  each  OCE. 
We  assigned  a  score  of  2  to  each  black  dot,  a  score  of  1  to  each  white  dot, 
and  a  score  of  0  to  each  "no  support."  The  sum  over  aU  the  scores  for  all 
OCEs  gives  the  total  military  contribution  for  each  program. 

Every  time  a  program  was  changed  in  the  formulation  of  some  overall 
plan,  the  software  prompted  the  person  making  the  modification  to 
explain  his  reasons.  These  comments  appear  tmder  the  Notes  column. 
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“Workspace”  Gave  Panelists 
a  Variety  of  Tools 

I  NDRI  ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 

Each  panelist  could 

•  edit,  duplicate,  and  delete  plans  on  his  private  scratch  pad 

•  share  technology  plans  with  fellow  panelists 

•  compare  two  plans’  individual  programs,  overall  budgets,  and  the 
military  capabilities  they  enable 

•  assess  the  contribution  a  plan  makes  to  future  military  capabilities 

Panelists  used  alternative  plans  as  exhibits  in  their 
discussions. 

Facilitators  helped  each  panel  reach  consensus  on  a  plan 

and  assess  its  military  contributions. 

wmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm  randi 


Each  panelist  had  a  workspace  on  the  server,  in  which  he  could 
manipulate  versions  of  the  JWS&T  Plan.  Panelists  could  edit,  duplicate, 
and  delete  plans  as  they  could  with  any  standard  word-processing 
document.  While  working  on  a  plan,  they  could  keep  it  private  on  their 
own  scratch  pad  and  then,  when  ready,  share  that  plan  with  other 
panelists.  The  typical  Information  Superiority  plan  contained  several 
hundred  programs,  so  we  provided  tools  that  allow  a  panelist  to  compare 
two  technology  plans  and  quickly  focus  on  the  specific  differences 
between  them.  Finally,  we  provided  the  panelists  with  tools  to  assess 
how  well  a  technology  plan  contributes  to  future  military  capabilities,  as 
we  discuss  below. 

Since  these  plans  were  online  along  with  the  discussions,  panelists  could 
use  the  plans  as  exhibits  in  the  discussions.  For  instance,  a  panelist  could 
propose  modifications  to  the  plan  and  post  them  for  tihe  other  panelists  to 
debate. 

With  help  from  the  facilitator,  each  team  agreed  on  a  single  version  of  the 
plan  for  each  team. 
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Step  3 — Assess  Performance  of  Plan: 

Panelists  Assessed  Plans  with  ABIS  “Water  Glasses 


III 

>-L  sh  2 

•B  wn 

o  S* 

IS® 

ill 


-  “I” 

■3  III 

i  ^ 

I 

3  iH 

ISi 


I  *‘g  ltT 

1  ,o 

1  ■  ^  a 
I  n 

E-t 

00 

;  s  o 

,  wl - 


w  m  a> 

gWrS 
w  O  § 

i  I  ^ 


30 


Set  Nomialization  Yelues 


Once  the  panelists  had  agreed  on  a  modified  technology  plan,  the  facilitator 
asked  them  to  move  to  Step  3,  in  which  they  assessed  how  well  the  United 
States  could  perform  the  urban-combat  mission  if  the  proposed  plan  were 
enacted. 

In  the  ABIS  study,  the  team  used  the  "water  glass"  method  to  assess  how  well 
each  Operational  Capability  Element  in  that  technology  plan  contributed  to 
future  military  capabilities.  Each  OCE  was  represented  by  a  partially  filled 
water  glass,  lire  more  full  the  glass,  the  greater  the  contribution  of  tihe 
technology  plan  to  tihe  OCE.  In  ABIS,  all  the  water  glasses  were  either  1/2  or 
3/4  fuU.  We  adopted  that  measure  for  tiiis  work,  and  adopted  those  values  as 
a  starting  point  for  the  discussion. 

The  ABIS  OCEs  were  identical  to  eight  of  the  entries  in  the  table  on  the  facing 
page;  within  the  JWS&T  Plan,  Information  Acquisition  was  added  as  another 
OCE.  In  ABIS,  the  acquisition  of  information  had  been  ruled  out  of  scope,  so 
that  the  study  became  focused  on  the  less-studied  C4  (command,  control, 
communications,  and  computers)  aspects.  We  arbitrarily  set  Information 
Acquisition  to  start  at  3/4. 

In  our  exercise,  we  asked  that  the  panelists  estimate  the  contribution  of  their 
entire  technology  plan  to  each  of  the  nine  Information  Superiority  OCEs 
interpreted  witlun  their  scenario  and  concept  of  operations.  As  in  ABIS,  we 
asked  for  that  estimate  on  a  0-to-l  scale.  This  chart  shows  the  page  the 
panelists  used  to  enter  their  judgments.  The  number  in  each  box  is  entered  by 
the  panelists.  The  corresponding  number  in  parentheses  is  the  ABIS  water- 
glass  height  for  that  OCE,  or  for  Information  Acquisition,  our  surrogate. 

In  an  associated  discussion  area,  panelists  debated  their  choices  imtil  their 
team  reached  a  consensus. 
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Against  Other  Military  Scenario 
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Email:  TeamOBfSJremd.  OK  Teclmdcal  Support:  tem?@r9M.  ok 


Up  until  this  point,  each  team  had  worked  separately  on  crafting  a 
technology  plan  for  a  particular  future  military  scenario.  Once  each  team 
had  completed  this  job,  we  gave  it  the  other  team's  scenario  in  order  to  test 
the  robustness  of  their  plan.  We  asked  Team  A  to  assess  how  well  its 
technology  plan,  designed  for  the  No-Waming  MRC  scenario,  would 
contribute  to  the  Urban  Combat  scenario.  Similarly,  we  asked  Team  B  to 
assess  how  well  its  technology  plan,  designed  for  the  Urban  Combat 
scenario,  would  contribute  to  the  No-Waming  MRC  scenario. 

The  figure  on  the  facing  page  shows  the  new  homepage  for  Team  B  at  the 
start  of  this  fourth  and  final  step.  We  allowed  Team  B  to  see  the  work  they 
and  Team  A  did  in  Steps  1  through  3.  Team  B  also  got  a  new  Step  4, 
asking  them  to  develop  concepts  of  operations  for  the  No-Waming  MRC 
scenario  and  to  assess  how  well  tiheir  Urban  Combat  technology  plan 
would  address  this  new  scenario.  It  is  important  to  remember  fiiat  this  is 
an  assessment  of  the  plan  loith  no  adjustments  for  the  new  scenario.  The 
search  for  a  plan  that  works  well  in  both  scenarios  comes  later  in  the 
HyperForum. 

Note  that  fiie  site  for  Team  B  changes  significantly  when  it  moves  to  Step  4 
fi:om  Steps  1  through  3.  (The  Team  A  site  changed  similarly.)  Changing 
the  site  over  time  is  an  important  feature  of  the  HyperForum.  It  allows  the 
panelists  and  facilitator  to  move  systematically  through  a  series  of  tasks 
and  the  information  supporting  fiiose  tasks. 
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Result  Is  Both  Plans  Assessed  Against 
Both  Military  Scenarios 


Plan  A 


Plane 


No-Warning  MRC  Urban  Combat 


RANDBBi 


This  chart  presents  a  graphic  summary  of  the  panels'  work  showing  how 
each  team  assessed  the  military  capabilities  that  would  derive  from 
successful  completion  of  their  S&T  plans.  Both  scenarios  used  each  of  the 
nine  Information  Superiority  OCEs  from  the  JWS&T  Plan.  These  OCEs  are 
Prediction  Planning  (ppp).  Integrated  Force  Management  (ifm).  Execution 
of  Time-Critical  Missions  (etcm),  Information  Acquisition  (ia),  Battlespace 
Understanding  (cbu).  Precision  Information  (pid).  Universal  Transactions 
Served  (uts).  Distributed  Environmental  Support  (des),  and  Assurance  of 
Services  (as). 

As  we  would  expect,  each  plan  works  best  for  the  scenario  it  was  designed 
for.  The  ratings  for  the  off-design  scenarios.  Urban  Combat  for  Plan  A  and 
No-Waming  MRC  for  Plan  B,  are  important  for  this  effort,  because  they 
form  the  basis  for  the  final  step;  interpolating  between  plans  in  a  search 
for  robustness. 

The  detailed  changes  have  an  immediately  apparent  logic.  Plan  A, 
developed  for  the  MRC,  had  the  largest  estimated  decrease  in  its  capability 
for  Battlespace  Awareness  (the  middle  three  OCE  estimates  above)  when 
facing  urban  combat.  For  instance,  Liformation  Acquisition  for  Plan  A 
drops  significantly  in  Urban  Combat  compared  with  the  MRC.  In  the 
latter  scenario,  tiiere  is  no  need  to  look  inside  buildings;  this  is  an 
operation  essential  in  urban  combat  but  not  in  an  MRC.  Similarly,  when 
facing  a  No-Waming  MRC,  Plan  B,  derived  for  urban  combat,  had  the 
largest  estimated  decrease  in  its  capability  to  provide  integrated  force 
management,  reflecting  the  larger  scale  of  coordination  needed  in  an  MRC. 
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Flow  of  Experimental  Planning  Process 
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Task  1: 

Task  2: 

Task  3: 

Task  4: 

Tasks: 


Create  Web  site  with 
specialized  tools  and 
content 

Convene  expert  panels 


Conduct  Exploratory 
Modeling  exercise 

Iterate  with  expert 
paneis 

Provide  decisionmakers  with 
plan  and  history 
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4.  TASK  3:  CONDUCT  EXPLORATORY 
MODELING  EXERCISE 


The  two  teams  in  our  expert  panel  produced  two  technology  plans,  each 
assessed  against  different  future  military  scenarios.  But  policymakers 
need  a  single  technology  plan  that  is  robust  against  multiple  futures.  In 
this  section  of  the  briefing,  we  describe  a  simple  application  of  the 
Exploratory  Modeling  ideas  to  suggest  how  the  two  teams'  plans  can  best 
be  combined  into  a  single,  robust  technology  plan. 

We  describe  how  we  can  automatically  generate  new  technology  plans  that 
are  mixes  of  the  programs  from  Plans  A  and  B  and  that  have  the  same 
overall  budget  as  Plans  A  and  B.  We  then  describe  how  we  can  estimate 
the  military  contribution  of  each  of  these  new  plans  in  our  two  military 
scenarios.  This  process  is  designed  to  produce  a  small  number  of 
candidate  technology  plans,  successful  in  both  military  scenarios,  that  in  a 
real  plaivning  process  would  be  given  back  to  the  expert  panels  to  help 
them  recommend  a  single,  robust  technology  plan. 
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Team  A  Modified  Existing  Programs; 
Team  B  Added  New  Programs 


Plan  A  Plan  B 


Three  programs  added 
($48  million) 

Eleven  programs 
modified  in  plan 
($1789  million) 

Two  programs  deleted 
($91  million) 


92  programs 
unchanged 
($402  million) 


Plan 


Budgets 


Ten  programs  added 
($1301  million) 

Six  programs  deleted 
($1312  million) 


101  programs 
unchanged 
($939  million) 


randhhi 


As  a  first  step  in  explaining  our  method  of  combining  technology  plans,  it 
is  useful  to  examine  in  more  detail  the  results  produced  by  our  expert 
panel.  This  slide  shows  a  budgetary  summary  of  how  the  two  teams 
modified  the  JWS&T  Plan.  The  differing  shaded  areas  in  the  bars  indicate 
the  very  different  responses  of  our  two  teams  to  their  respective  scenarios. 

The  Information  Superiority  portion  of  the  JWS&T  Plan  had  a  total  budget 
of  $2.2  billion.  As  shown  in  the  figure.  Team  A  added  three  new  programs 
costing  a  total  of  $48  million,  modified  eleven  programs  accounting  for 
$1789  million,  deleted  two  programs  totaling  $91  million,  and  left 
unchanged  92  programs  totaling  $402  million.  Team  B  added  ten  new 
programs  costing  $1301  million,  deleted  six  programs  totaling  $1312 
million,  and  left  101  programs  totaling  $939  million  unchanged.  Team  B 
did  not  modify  any  programs. 

The  contrast  between  these  two  sets  of  results  makes  sense  in  light  of  the 
different  scenarios  given  to  the  two  teams.  Team  A's  No-Waming  MRC 
scenario  was  close  to  the  dominating  MRC  scenario  considered  by  the 
government  in  creating  the  original  JWS&T  Plan.  Accordingly,  Team  A 
could  focus  on  modifying  existing  programs.  Team  B's  Urban  Combat 
scenario  was  very  different  from  the  situations  considered  in  ABIS  and  in 
tiae  Information  Superiority  section  of  the  JWS&T  Plan.  The  panelists 
responded  by  creating  new  programs  designed  especially  for  these  new 
circumstances. 

As  an  aside,  it  is  interesting  to  note  the  relatively  small  budget  for  the  large 
number  of  imchanged  programs.  This  owes  to  the  fact  that  many 
programs  in  the  original  JWS&T  Plan,  especially  those  beyond  the  5-year 
budget-planning  cycle,  had  no  budget.  Consequently,  there  was  no 
incentive  to  remove  or  change  them. 
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Changes  Made  by  Teams  A  and  B 


Team  A 

Additions  to  Plan 

Rapid  Terrain  Visualization  -  Mod 
Adaptive  Force  Package  Tailoring 
Joint  Theater  Information  &  Spectrum 
Dominance 

Modified  Programs 

Distributed  Situation  Assessment 
Robust  Tactical/Mobile  Networking 
Joint  C4I  for  Rapid  Force  Projection 
Precision  Guided  Mortar  Munition 
IW  [Information  Warfare]  Battle  Management 
Survivabie  Armed  Reconnaissance  on  the  Digital 
Battlefield 

Rapid  Battlefield  Visualization 
Battlefield  Awareness  &  Data  Dissemination 
Operator  Intelligence  Interface 
Small  Satellite  SAR  [Synthetic  Aperture  Radar] 
Counter  Camouflage,  Concealment,  &  Deception 
Deleted  from  Plan 
Precision  Rapid  Counter  Multiple  Rocket 
Launcher 

Rotocraft  Pilot  Associate 


Team  B 

Additions  to  Plan 

MOUT  [military  operations  on  urban  terrain] 
Command,  Control,  Comm,  Computers  and 
Intelligence 
MOUT  Survivability 
MOUT  Modeling  and  Simulation 
Full-Scale  MOUT  Demo 
Urban  Combat  Training  ACTD 
Netted  Sensor  ATD 
Mirage 

Context-based  Information  Distribution  2 
Robotic  Sensors 

Smart  Assistant  for  the  Infantryman 
Deleted  from  Plan 
Precision  Rapid  Counter  Multiple  Rocket 
Launcher 

Precision  Guided  Mortar  Munition 
GEODSS  [Ground-Based  Electro-Optical  Deep- 
Space  Surveillance  System]  Upgrade 
Small  Satellite  SAR 
Wide  Area  Tracking  System 
Context-Based  Information  Distribution 

RAND 


This  chart  names  the  specific  programs  added,  modified,  and  deleted  by 
our  expert  panels.  All  the  modifications  made  by  Team  A  had  to  do  with 
the  budget,  as  opposed  to  the  substance,  of  the  programs  in  question. 

Team  A  increased  or  decreased  the  budgets  of  programs  to  provide  more 
or  fewer  capabilities.  They  did  not  modify  or  create  new  programs  to 
provide  a  significantly  different  capability.  Many  of  the  programs  added 
by  Team  B  were  taken  from  other  parts  of  the  JWS&T  Plan — specifically, 
the  Military  Operations  in  the  Urban  Terrain  JWCO — ^rather  than  being 
created  from  scratch.  Important  exceptions  are  Team  B's  Mirage  program, 
which  we  discussed  earlier,  and  their  Netted  Sensor  ATD,  Robotic  Sensors, 
and  Urban  Combat  Training  ACTD. 
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Interpolating  Technology  Plans 


■  NDRIt 


Plan  A 


Intermediate  Plan 


Plane 


Information  Superiority 
OCEs 


Information  Superiority 
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Mirage 
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If  we  knew  with  certainty  that  the  future  would  look  like  the  scenarios  given  to 
Team  A,  we  would  choose  Plan  A  as  our  new  JWS&T  Plan.  If  we  knew  with 
certainty  that  the  future  would  look  like  the  scenario  given  to  Team  B,  we  would 
choose  Plan  B.  But  since  we  do  not  know  the  future,  we  want  to  examine 
whether  some  composite  of  Plans  A  and  B  is  reasonably  robust  against  both 
scenarios. 

Accordingly,  we  developed  an  automated  routine  for  interpolating  two 
technology  plans,  as  shown  in  this  chart.  An  automated  plan  is  not  really 
necessary  with  only  two  plans;  however,  we  wanted  to  demonstrate  our  ability 
to  expand  in  the  future  to  many  dozens  or  hundreds  of  plans  to  cover  many 
more  futures. 

We  start  by  noting  that  each  plan  has  programs  rmique  to  that  plan  and 
programs  common  to  both  plans.  The  total  budget  for  the  programs  unique  to  A 
is  the  same  as  the  total  budget  for  the  programs  unique  to  B,  since  the  budgets  of 
the  plans  common  to  A  and  B  are  the  same  and  both  plans  have  the  same  total 
budget. 

Any  plan  intermediate  between  Plans  A  and  B  will  have  all  the  programs 
common  to  A  and  B.  The  remainder  of  the  budget  is  then  used  to  buy  programs 
rmique  to  A  or  B  for  the  intermediate  plan.  For  instance,  an  intermediate  plan 
that  is  75  percent  A  and  25  percent  B,  uses  75  percent  of  its  "Unique  to  A  or  B" 
budget  to  buy  programs  unique  to  A  and  25  percent  of  the  budget  to  buy 
programs  tmique  to  B.  (See  the  example  on  page  40.) 
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Finally,  we  need  to  determine  the  military  contribution  of  the  intermediate  plan. 
As  shown  in  the  chart,  each  individual  program  contributes  to  some  OCEs,  as 
measured  by  the  white- /black-dot  weighting  discussed  above.  Assuming  there 
are  no  important  synergies®  among  programs,  we  sum  the  contributions  of  each 
program  for  each  OCE  to  estimate  the  military  contribution  of  the  intermediate 
plan. 

Note  that  this  process  does  not  distinguish  contributions  in  different  scenarios. 
Ratiier,  a  contribution  of  some  weight  to  some  OCE,  say.  Information 
Acquisition,  is  the  same  in  any  scenario.  Although  the  contribution  of  any  one 
program  is  always  only  a  few  percent  of  the  total  contribution  for  any  OCE  and 
does  not  have  a  huge  effect,  it  has  some  distorting  consequences  for  the  method. 
These  consequences  are  discussed  in  the  chart  after  next  and  could  be  changed 
with  a  more  elaborate  weighting  methodology.  We  adopted  this  interpolation 
process  because  it  was,  literally,  the  closest  analogy  that  we  could  find  to  the 
subjective  JWS&T  methodology.  That  methodology  does  not  discuss 
contributions  in  different  scenarios;  rather,  it  presents  the  entire  plan  and 
contributions.  Additionally,  even  this  method  is  time-consuming,  so  elaboration 
in  this  experiment  was  implausible. 


^Synergies  are  enhancements  produced  in  one  program  by  the  presence  of  another.  For  example, 
there  are  S5mergies  between  improved  long-range  weapons  and  improved  long-range  battlefield 
surveillance,  since  the  former  are  more  useful  if  a  military  force  knows  what  is  in  the  battlefield 
to  shoot  at.  Generally,  there  are  synergies  among  technology  programs,  although  most  often 
they  are  not  considered  explicitly  in  most  analyses,  including  the  JWS&T  Plan  and  ours. 
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Interpolation  with 
75%  Plan  A  and  25%  Plan  B 


INDRII 


PLAN  A  PROGRAMS  Score  Budget 

Robust  Tactical/Mobile  Networking  5  $61.5 

Context’ Based  Information  Distribution  4  $0.0 

Rapid  Terrain  Visualization  -  Mod  4  $4.5 

Distributed  Situation  Assessment  4  $30.0 


Joint  Theater  Information 

&  Spectrum  Dominance  4 

Joint  C4I  for  Rapid  Force  Projection  4 

Battlefield  Awareness 

&  Data  Dissemination  4 

Adaptive  Force  Package  Tailoring  3 

IW  Battle  Management  3 

Wide  Area  T racking  System  2 

GEODSS  Upgrade  2 

Counter  Camouflage, 

Concealment,  &  Deception  2 

Operator  Intelligence  Interface  2 

Rapid  Battlefield  Visualization  2 

Small  Satellite  SAR  2 

Survivable  Armed  Reconnaissance 
on  the  Digital  Battlefield  0 

Precision  Guided  Mortar  Munition  O- 


$35.0 

$110.0 

$225.7 

$8.0 

$33.0 

$3.0 

$6.6 

$25.0 
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$125.0 
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PLAN  B  PROGRAMS  Score  Budget 

Full-Scale  MOUT  Demo  11  $140.0 
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The  interpolation  routine  requires  that  the  programs  unique  to  each  plan 
be  prioritized.  We  employed  a  simple  scheme  of  ranking  the  programs  in 
eadr  plan  according  to  their  military  contribution.  We  broke  ties  between 
programs  having  equivalent  military  contributions  by  ranking  those 
programs  with  smaller  cost  higher  than  programs  with  higher  cost. 

This  is  obviously  a  crude  prioritization  scheme,  although  it  is  closely  based 
on  the  assessments  used  in  the  ABIS  project.  In  future  work,  it  would 
certainly  be  possible  to  employ  other  schemes.  It  is  also  possible  to 
compare  the  technology  plans  resulting  from  several  prioritization 
schemes,  so  that  the  panelists  need  not  come  to  a  consensus  on  any  single 
ranking  of  the  programs. 

Note  too  that  synergies  between  programs,  or  dependencies  among  them, 
can  be  captured  with  elaborations  of  this  method.  Although  we  built  tools 
that  can  capture  such  relationships — allowing,  for  example,  one  program's 
failure  to  "force"  the  degradation  of  another — ^it  was  clear  that  our  panels 
had  neither  the  time  nor  the  expertise  to  use  such  linkages.  Indeed,  such 
relationships  are  also  largely  ignored  in  other  planning  methods  we  have 
seen. 

These  other  methods  might  be  able  to  capture  and  to  display  the 
implications  of  such  relationships,  creating  new  insights  for  S&T  planning. 
For  now,  however,  that  ability  remains  a  speculation. 
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Mix  of  40-70%  Plan  B 
Gives  Most  Robust  Overall  Plan 


—  NDRI 


No-Warning  MRC  Urban  Combat 


RAND 


Next,  we  foxind  the  most  robust  combination  of  die  technology  plans 
produced  by  our  expert  panel.  The  results  of  our  analysis  of  the  combinations 
is  shown  in  this  chart. 

The  left  and  right  panels  show  how  a  variety  of  intermediate  plans  perform  in 
the  No-Waming  MRC  and  Urban  Combat  scenarios,  respectively.  Each  curve 
in  the  panels  represents  the  performance  of  one  Information  Superiority  OCE, 
measured  along  the  0-to-l  water-glass  scale,  as  a  function  of  the  fraction  of  the 
"Unique  to  A  or  B"  budget  allocated  to  Plan  B  programs.  For  instance,  the 
Information  Acquisition  (ia)  OCE  in  the  Urban  Combat  scenario  has  a  score  of 
0.3  when  the  entire  budget  is  allocated  to  Plan  A  programs  and  increases 
monotonically  to  a  score  of  0.65  when  the  entire  budget  is  allocated  to  Plan  B 
programs.  This  behavior  is  not  surprising,  since  the  technologies  needed  to 
locate  enemy  forces  are  quite  different  for  an  MRC  and  for  Urban  Combat. 

This  chart  shows  us  that  the  technology  plans  most  robust  against  our  two 
military  scenarios  are  those  in  which  40  to  70  percent  of  the  budget  is  allocated 
to  Plan  B  programs.  Note  that  below  40  percent,  a  technology  plan  performs 
relatively  poorly  in  the  Urban  Combat  scenario.  Above  70  percent,  a  plan 
provides  relatively  poor  performance  in  the  No-Waming  MRC  scenario. 

This  represents  the  final  step  in  our  demonstration  project.  In  a  real-world 
planning  process,  this  result  would  be  given  the  panelists,  who  would  then 
use  this  information  as  a  starting  point  to  craft  an  S&T  plan  robust  against 
these  two  military  futures. 
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Note  that  the  OCEs  for  the  No-Waming  MRC  scenario  improve  when  about  10 
percent  of  the  Plan  A  budget  is  allocated  to  Plan  B  programs  and  for  the  Urban 
Combat  scenario  when  about  10  percent  of  the  Plan  B  budget  is  allocated  to 
Plan  A  programs.  The  reason  is  that  Plans  A  and  B  both  have  a  small  number 
of  programs  with  negligible  contribution  to  the  Information  Superiority  OCEs. 
Our  interpolation  scheme  replaces  these  programs  with  the  highest-scoring 
programs  of  the  other  plan,  and  thus  increases  the  overall  performance  of  the 
intermediate  plans. 

This  is  one  example  of  a  potential  distortion  coming  from  our  simple  weighting 
method.  Presumably,  the  new  programs  would  contribute  most  in  the  other 
scenario,  and  so  give  less  of  a  boost  when  they  are  added.  This  effect  is  also,  in 
part,  due  to  difficulties  in  the  then-current  JWS&T  Plan,  which  included 
programs  having  low  contributions. 
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Panelists  Judged  the  Exercise  a  Success 


mmm  injury 

Even  without  much  prior  Web  experience,  most  panelists 
participated  without  difficulty: 

•  Most  required  several  hours  to  come  up  to  speed 

•  Most  were  heavily  engaged  in  discussions 

Panelists  enjoyed  flexibility  of  asynchronous  meetings 

•  But  still  needed  to  devote  significant  time  to  effort 

Compared  with  face-to-face  meetings,  HyperForum  brought 
more  information  into  discussions 

Facilitator  was  necessary 

ffliilllllllllWBliiliMlllillM^  RANOam 

5.  CONCLUSIONS 


After  tine  experiment,  we  interviewed  the  participants  to  learn  what  they 
liked  and  did  not  like  about  the  experiment. 

In  general,  the  panelists  were  pleased.  AH  had  been  able  to  access  the  site, 
learn  how  to  use  it,  and  participate  in  the  discussions.  All  were  impressed 
by  being  able  to  conduct  detailed,  informative,  asynchronous  discussions. 
Many  reported  that  their  other  commitments  would  have  prevented  them 
from  participating  in  the  exercise  if  they  had  been  required  to  physically 
attend  a  series  of  meetings  at  a  particular  place  and  time.  The  participants 
also  reported  that  the  online  format  provided  easier  access  to  relevant 
information  than  most  face-to-face  meetings  could  provide. 

Nonetheless,  many  panelists  reported  that  they  spent  many  hours 
examining  the  material  on  the  site  and  composing  their  own  postings. 
Asynchronous  does  not  imply  that  little  time  was  needed,  only  that  the 
schedule  for  that  time  was  flexible.  The  facilitators  spent  even  more  time 
than  did  panelists  in  sifting  and  combining  the  messages  in  a  way  that 
structured  the  discussion.  In  general,  this  HyperForum  did  not  substitute 
for  human  knowledge  and  insights,  but  provided  a  different  and,  in  some 
ways,  a  superior  way  to  approach  a  goal  cooperatively. 
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Project  Demonstrated 
Some  Key  Capabilities 


Technology  plans  can  provide  and  draw  on  more 
sources  of  information.  Plans  are 


•  Databases  that  can  be  searched  and  manipulated 

•  Nodes  linked  to  other  relevant  sources  of  information 


•  Forums  with  inputs  and  buy-in  from  many  stakeholders 


Technology  plans  can  embrace  uncertainty: 


•  Portfolios  that  take  advantage  of  fortuitous  opportunities 
and  hedges  against  adverse  surprises 


•  Adaptive  processes  designed  to  respond  as  new 
information  becomes  available 


RAND  I 


Overall,  this  experiment  demonstrated  many,  but  not  all,  of  the  key 
capabilities  we  believe  a  method  combining  HyperForum  and  Exploratory 
Modeling  can  offer.  First,  it  showed  that  large  databases  can  be  searched 
and  manipulated  online  by  individuals  with  the  computer  skills  of  those 
who  typically  serve  on  DoD  expert  panels.  Moreover,  these  databases  can 
be  linked  to  yet  more  sources  of  information  online,  enabling  the  panelists 
to  gadier  much  relevant  information  quickly  without  the  delays  inherent  in 
a  face-to-face  meeting. 

Additionally,  the  method  did  demonstrate  a  capability  to  fashion  plans 
that  show  robustness  to  multiple  (two)  futures.  That  part  of  the 
methodology  is  easily  expanded  to  as  many  scenarios  as  panelists  can 
consider. 

We  did  not  show  that  diverse  stakeholders  would  buy  into  this  method. 
Our  panelists  did,  but  they  were  not  tmder  tire  same  sort  of  institutional 
pressures  that  real-world  panelists  will  always  feel.  We  also  did  not 
pursue  Exploratory  Modeling  into  the  realm  of  adaptive  strategies — that 
is,  strategies  explicitly  designed  to  evolve  over  time  in  response  to  new 
information;  rather,  this  method  would  simply  be  used  to  revise  plans  as 
needed  (yearly  for  the  JWS&T  Plan).  Both  HyperForum  and  Exploratory 
Modeling  could  be  a  focus  for  future  work. 

Additionally,  we  fotmd  one  insight  not  mentioned  in  the  slides.  The 
weighting  plan  of  the  established  methodology  in  the  JWS&T  Plan 
discriminates  against  large  programs.  Essentially,  no  program  contributes 
to  more  than  four  OCEs,  and  not  "strongly"  even  to  that  many. 
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Consequently,  no  existing  program  has  a  weight,  in  our  scheme,  of  more 
than  9  units.  Conversely,  every  program  should  have  some  contribution, 
which  is  a  minimum  of  1  unit.  Thus,  the  dynamic  range  of  different 
programs'  contribution  to  (all)  OCEs  was  about  10. 

In  contrast,  the  budgets  of  the  differing  programs  varied  by  well  over  a 
factor  of  100,  which  meant  that  it  was  always  better  to  trade  off  a  large 
project  for  a  host  of  smaller  ones,  from  another  plan  or  from  some  variant 
plan  being  considered  for  just  one  scenario.  For  example,  the  Small 
Satellite  Synthetic  Aperture  Radar  program,  which  was  in  the  plan  for  $660 
million  and  had  an  OCE  contribution  of  2  units,  could  easily  be  traded  off 
for  a  great  number  of  modeling  and  simulation  programs,  which  cost, 
typically,  a  few  million  dollars  and  gain  a  contribution  of  1  or  2  xmits. 

It  is  important  to  note  that  this  is  not  a  flaw  in  our  methodology  or 
weightings,  which  we  set  arbitrarily.  Rather,  the  problem  is  inherent  in  the 
JWS&T  Plan  itself,  which  contains  such  a  wide  range  of  program  sizes  that 
the  judgments  of  program  importance  (the  weightings  or,  in  the  j[WS&T 
Plan,  sets  of  "dots")  are  overwhelmed.  Our  computerized  method  of 
comparing  these  programs  forced  us  to  confront  this  issue,  which  is  still 
present  in  the  JWS&T  Plan  as  it  exists  today. 

FimdamentaUy,  if  the  estimates  of  operational  importance  in  the  JWS&T 
Plan  are  taken  seriously,  they  lead  to  the  cutting  of  large  programs.  The 
leadership  of  OSD  should  consider  either  modif5dng  those  estimates  of 
importance  to  allow  them  to  express  a  much  greater  impact  from  different 
efforts,  or  else  restrict  the  JWS&T  Plan  to  consideration  of  only  programs  of 
roughly  similar  costs. 
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Methods  Offered  Interesting  Advantages 


•  Allow  asynchronous,  remote  involvement 

-  Lower  some  costs 

-  Broaden  participation 

•  Provide  traceability  through  discussion 

-  Generate  recognized  support  for  ideas 
-Affect  deliberations 

•  Create  more-robust  plan 

-  Address  multiple  futures  more  easily 

•  Articulate  robustness  to  outside  audiences 

mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm  rand 


Returning  to  the  consequences  of  our  experiment,  we  found  many  of  the 
practical  advantages  we  anticipated.  The  combined  method  does  allow 
more  people  to  participate  and  even  lowers  travel  costs.  It  provides  a 
usable  traceability  so  that  decisions  or  recommendations  can  be  fully 
understood  by  otihers  coming  to  the  process  later.  Combining  different 
plans  is  possible  and  creates  a  more  robust  hybrid.  Finally,  that  robustness 
can  be  commimicated  with  some  flexibility  to  outside  audiences. 
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Next  Steps 


Apply  methods  to  real-world  planning  process: 

•  Assess  robustness  of  DDR&E’s  current  JWS&T  Plan 

•  Assist  other  federal  agencies  with  S&T  planning 

•  Help  private-sector  firms  craft  R&D  portfolios 

Methods  most  useful  when  there  is 

•  Desire  to  create  portfolios  of  S&T  investments  robust  against 
extreme  uncertainty 

•  Commitment  at  the  top  to  augment  current  planning  with  new 
methods 

•  A  large  number  of  stakeholders  in  planning  process 

•  Existing  planning  process,  based  on  expert  opinion,  in  place 

RANDHBB 


These  S&T  planning  methods  are  ready  to  use  as  part  of  a  real-world 
planning  effort.  From  our  limited  experience,  it  is  hard  to  generalize  on 
the  applicability  of  the  methods  used  in  this  proof  of  concept.  However, 
we  think  that  these  methods  will  be  most  useful  when  several  conditions 
are  met.  First,  there  must  be  a  need  to  craft  an  S&T  investment  portfolio 
that  is  robust  in  the  face  of  major  xmcertainties.  Without  imcertainty, 
traditional  planning  metiiods  will  probably  suffice.  Second,  there  must  be 
a  commitment  within  the  organization  to  exploring  new  methods  to 
augment  current  planning  methods.  Third,  the  methods  will  be  most 
useful  if  multiple  stakeholders  are  involved  in  the  planning  process;  a 
small  group  could  just  as  easily  make  their  plans  face-to-face.  Finally,  as 
currently  configured,  our  combined  method  is  designed  to  augment  an 
existing  planning  process  based  largely  on  expert  opinion.  In  the  future, 
however,  we  hope  to  expand  the  methods  to  combine  expert  opinion  with 
other  types  of  information. 

As  an  obvious  first  step  in  such  expansion,  these  S&T  planning  methods 
could  be  used  as  part  of  the  next  planning  cycle,  to  assess  the  robustness  of 
some  portion  of  the  next  JWS&T  Plan.  If  the  experience  is  positive,  the 
methods  could  then  be  used  to  address  the  full  plan. 

This  combined  method  could  also  be  used  for  other  S&T  planning 
activities  within  the  Department  of  Defense — for  example,  those 
conducted  by  the  individual  Services.  In  addition,  other  federal  agencies 
that  build  S&T  plans,  such  as  NASA  or  the  Department  of  Energy,  might 
also  find  these  methods  useful.  For  instance,  ihe  National  Science  and 
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Technology  Council  might  use  these  tools  to  build  consensus  among 
scientists,  industry  representatives,  and  environmentalists  on  a  national 
technology  agenda  for  dealing  with  the  threat  of  climate  change.  Finally, 
private  industry  might  also  use  these  methods  to  craft  technology  research 
portfolios  well  balanced  between  protection  of  their  current  markets  and 
the  challenge  of  creating  new  ones. 
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APPENDIX 


By  GEAIWE  ROSENBERG'  : 

:  :  '  r'..  •  ^  V 

The  year  is  2050,  The  placel'Earth. 
The  possibilities:  a  chaotic  world  at 
the  mercy  of! warring  factions;*  a 
world  in  which  sustainable  develop¬ 
ment  is  mediated  by  governmental 
action;  a  world  in.  which  technology 
has  eliminated  the  threats  posed  by 
global  warming,  populadon  growth 
and  diminished  natural'  resources. 
Which  future  is  in  store  and  what,  if 
anything,  can  or  should  i,be'  done 
about  it?  ■  .•  •  -.j  .  ‘ 

These  quesUons  were  wllectively 
addressed  by  environmentalists,- 
economists,  academicians 'and  policy 
makers.  But^while  the  Rand  Corpo- 
raiipn  jand  the  World  Resources  In¬ 
stitute  helped  set  up  the  debate,  the 
.  arena  was  not  a  conference  room. 

Instead,  it  was  Hyper  Forum,  a 
site  in  cyberspace  where  experts 
convened  earlier  this  year  to  "grap¬ 
ple  with  the  range  of  plausible  fu¬ 
tures,"  said  Robert  Lempert,  a  sen¬ 
ior  scientist  at  Rand  who  helped  de¬ 
velop  Hyper  Forum  along  with  oth¬ 
ers  at  Rand,  the  -World  -Resources 
institute,  the  Markle  Foundation  and 
Bruce  Murray,  a  professor  of  plane¬ 
tary  science  and  geology  at  the  Cali¬ 
fornia  Institute  of  Technology  and 
Hyper  Forum  projecCleader. . 

At  Rand,  Dr.  Lem^n  said,  “We 
still  have  our  tank,”  But  the  notion  of 
a  tankless  “think  tank"  is  not  an 
oxymoron  —  not  in  cyberspace  at 
any  rate.  Experts  in  fields  as  diverse . 
as  politics,  economics^  the*  environ-';, 
ment,  law  and  business  are  choosing 
the  Internet  as  ^e  most  efficient- 
forum  for  policy  research.  ■  — 

A  case  in  point  is  the  Cyberspace  • 
Law  Institute,  an  Internet-based  re-' 
search  group  devoted  to  promoting  • 
discussion  on  the  law  of  cyberspace. 

Increasingly,  according  to  Maggie 
Powell,  electronic  publishing  man-" 
ager  at  the  World 'Resources  Insti-  ' 
tute  in  Washington,  the  Internet 
“gives  everyone  the  opportunity  to 
be  able  to  contribute  their  thoughts 
without  leaving  their  desks."  . 

Long-distance  collaboration  has 
been  an  organic  part  of  the  Internet 
since  it  was  incubated  in. academic  . 
and  Government  circles  in  the  1980’s. 
Businesses  have  been  built  around 
software,  like  Lotus  Notes,  which  en- ' 
hances.  such -coll  aboraUon. 

-  But  now  the  technolo'gy  .that  arose 
from  an  academic  _^viron ment  is  ; 
’  reluming  in  a  new»  guise:  as  a  tool  for 
research  centers  to  reorganize  poli¬ 
cy  discussions,  build  agreements 
among  specialists  and  reach  out  for 


new  voices  and  new  perspectives. 

.  Hyper  Forum  provides  a  clear  ex-, 
ample  of  how  the  Internet  is  chang¬ 
ing  the  way  research  centers  oper- 
ate..“The  idea  is,  a  forum  is  a  place 
where  people  come  together  to  de¬ 
bate,”  Dr.  Lempert  said.  Group  soft¬ 
ware  allowing  people  to  collaborate 
on  complex  issues  is  only  one  compo¬ 
nent  of  Hyper  Forum.  It  contains 
some  of  the  features  of  software  like 
Lo*tus  Notes,*  but  adds  information, 
stnicture  and  methodology. 

In  contrast  to  the  face-to-face 
meetings  that  research  centers  have 
■traditionally. held,  Hyper  Forum  al- . 
lows  “more  stakeholders,”  or  policy 
makers,  to  participate  and  can  reach 
people  who  otherwise  might  not  con-  . 
tribute.  Dr.  Lempert  said. 

•  Opening  .'the  doors  when  there  is 
free  food  for  thought,  however,  can 
draw  crowds.  David  Post,  a  law  pro-, 
fessor  who  co-founded  the  Cyber¬ 
space  Law  Institute,  discovered  this 

'virtual  reality  when  his  institute 
played  host  to  a  cyberconference  for 
nonlawyers  on  the  law  of  cyberspace. 
According  to  Professor  PosL  more 
^an  20,000  people  participated  via  E- 
mail.  "There’s  just  not  enough  time 
in  the  universe  to  deal  with  that,"  he 
said,  and  not  all  of  the  queries  could 
be  answered.,'  .  ■*•  * 

•  "The  problem  that  everyone  is  try-  • 
ihg  to  .deal  with  with  the  Web  is  the  - 
information  glut,!’  which  "degrades  •: 

'the  usefulness  of  the  information^’ 
iDr.  Lempert  observed.  To  resolve 
■  ‘this  problem,  he  said.  Hyper  Forum 
uses  a  moderator  to  guide  the  discus¬ 
sion.  ./ 

.  .Roger  Hurwitz,  research  scientist 


One  problem:  on¬ 
line  conferences 
a'ttract  too  many 
attendees.  ’  • 


at  the  Massachusetts  Institute  of 
Technology,  predicted  that  “then^ 
for  collaboration  and  managing  the 
complexity  of  collaboration”  is  going 
to  increase. 

Dr..  Hurwitz  and  John  Mallery, 
also  an  M.I.T.'  research  scientist, 
have  developed  software  for  wide- 
area  collaboration.  "Wide-area  col¬ 
laboration  is  the  idea  of  letting  ev¬ 
eryone  who  is  relevant  within  the 
enterprise  work  on  the  material,” 
Dr.  Hurwitz  explained.  His  software 
manages  and  directs  participation. 

The  'Virtual  Institute  of  Informa¬ 
tion,  a  search  and  link  network  for 
telecommunications,  cybercommun¬ 
ications  and  mass  media  reSaxch 
operated  by  the  Institute  for  Tele- 
Information  at  Columbia  University 
has  been  host  to"  cyberconferences. 
Topics  have  included  cryptography, 
electronic  publishing  and  telecom¬ 
munications  law,  said  Carla  Legen¬ 
dre,  communications  program  man¬ 
ager.'  .*.• 

Eli  Noam,  professor  of  finance  and 
economics  at  Columbia  and  director 
of  the  Institute  for  Tele-Information, 
has  experience  with  radio  and  video 
Internet  communications.  “There’s 
a  certain  aspect  at  this  point  of  'Look 


Ma,  no  hands,’ ^ he.'- said V of ”thel 
emerging  technologies'  But,  he''add| 

'  ed,  what  distinguishes  the  Internet 
that,  “you  have  the  ability'  to  estab^ 
lish  very  rapidly  shifting  groups  of^ 
people  to  form  ad  hoc  collaborative 
arrangements.”  .  \ 

The  Internet  is  providing  "a  better! 
informed  and  better  expressed  for¬ 
um  for  deliberative  discourse,”  Dr.* 
Murray  said,  ad^g:  “I  think  the' 
Web  is  an  amazing  revolution.  I  think* 
it’s  a  millennial  change  and  not  just  a 
decennial  or  centennial  one.”.;  .  (* 

While  Internet  users  can  chat  with, 

.  each  other  in  real  time  on  line,  Dr^ 
Murray  believes  that  electronic  cof^ 

•  respondence  —  which,  while  quicl^i 
allows  participants  a  chance  to  think 
before  they  write  —  is  the.  key  t§ 
collaboration  ■  on  complex  projects'^ 
Dr.  Murray  compares  use  of  the  Nef 
for  “deliberative  discourse”  :  tdj 
Thomas  Jefferson’s  correspondence; 
through  letter  writing  in  that  individr 
uals  can  take  the  time  to  analyze  a’ 
complex  situation  and  carefully 'conf.' 
struct  and  communicate,  their,  owii., 
thoughts,  reactions  or  •  additions.? 
However,  unlike  the  painfully  slow] 

!  process  of  written  correspondence" 
between  two  authors  in  the  days  be-' 
fore  the  locomotive,  the  Internet  alfl 
lows  almost  instantaneous  .and  si-J 
multaneous  access,  to  up-to-the^** 
minute  updates  of  huge,  amounts  of 
information  by  multiple  correspond- » 
ents  in  a  medium  that  unlike  the; 
penned  word;  “is  particularly  well; 
suited  for  visualizing  things.”  .  / 

In  addition,  he  said,  while  huge  , 
amounts  of  information  have  previ-‘! 
ously  been  organized  “in  a  serial 
sense,”  alphabetically,  or  by  num-' 
ber,  information  on  the  Net  “is  de-^ 
fined  only  in  connection  with  other; 
information”  which  is  “much  more,? 
the  way.  the  human  brain  works  — 

•  pattern  recognition.”  '*  ■  .  i 

But  while  the  Net  may  be  changing 
the  mode  of  collaboration  at  re-*; 
search  centers  ^d  elsewhere,  even] 
Internet  aficionados  agree  that  actu-^ 
al  meetings  are  not  obsolete.. 

Charles  M.  Firestone,  director  of* 
the  Aspen  Institute's  communica-- 
'■  tions  and  society  program,  said  the  { 
Internet  “affects  almost  every  level  ^ 
of  what  we  do,"  .  » 

•  But,  while  he  concedes  that  the ! 
World  Wide  Web  allows*  for  “more  ] 
.  people  to  participate’!  in  discussions.  * 
arid  “eliminates  physical  and  social  j 
:  barriers”  to  collaboration,-  “what  5 
:  you  really  want  to  do  is  have  the"; 
people  whom  you  want  to  resolve  the  : 
problem  at  the  table,”  he  said.  “Peo- 1 
pie  still  want  to  meet  face  to'  fac^” 


SOURCE:  "Internet  Changing  How  Research  Centers  Work,"  by  Geanne  Rosenberg,  and  illustration  by  Tom 
Bloom  are  reprinted  from  The  New  York  Times,  June  9, 1997,  p.  D5.  Copyright  ©  1997  by  The  New  York  Times 
Co.  Reprinted  by  permission. 


49 


